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Abstract 

Veterans and military service members enter the classroom with valuable life and leadership experience; 
however, transitioning to student life represents unique challenges. Like the larger veteran population, 
student veterans may bring to campus the negative aftereffects of their combat experiences in the form of 
post-traumatic stress, substance abuse, depression, and thoughts of suicide. To better support and retain 
the nation’s 950,000 student veterans, higher-education institutions are beginning to train faculty and staff 
to more effectively understand and meet the needs of student-veteran populations. This study examines 
the impact of a new and innovative game-based virtual training simulation where users role-play with 
emotionally responsive virtual student veterans to understand the unique value veterans bring to campus, 
the obstacles they face in their pursuit of a college degree, effective tactics for managing challenging 
conversations, and the best practices for connecting student veterans exhibiting signs of psychological 
distress with appropriate support services. Longitudinal data showed significant (p < .01) and sustained 
increases in learners’ military cultural competency (which included knowledge about the common 
challenges facing the student veterans), and preparedness to manage classroom discussions around 
veteran-sensitive issues, to talk with student veterans about their military service, and if necessary to 
make a referral to the appropriate campus office. Data also showed significant (p < .01) and sustained 
increases in gatekeeper skills, which included the learners’ preparedness, likelihood, and self-efficacy to 
help a student veteran in psychological distress, which resulted in increased referrals to support services. 
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Introduction 


Veterans and military service members represent an important portion of the higher-education 
population, with over 1 million veterans using their education benefits since 2008 (Department of 
Veterans Affairs, 2014). They share many attributes of other nontraditional populations, such as being 
adult learners and married college students; however, their experiences make them different from both 
traditional and nontraditional students. For example, student veterans and military service members bring 
a wealth of unique military experiences to the classroom due to their interaction with foreign cultures and 
the ensuing world perspective that can lead to more robust classroom discussions (Ackerman, DiRamio, 
& Mitchell, 2009). In addition, soft skills developed through military service, such as goal setting, time 
management, and discipline, can aid student veterans in their studying and academic performance. 
However, student veterans may also bring to campus the negative aftereffects of their combat experiences 
in the form of post-traumatic stress (PTS), depression, and thoughts of suicide, as reported in the larger 
post-9/11 veteran population (Hoge, Auchterlonie, & Milliken, 2006; Milliken, Auchterlonie, & Hoge, 
2007; Lapierre, Schwegler, & LaBauve, 2007). 

The prevalence of psychological distress among student veterans has not been thoroughly 
investigated or cataloged; however, several studies on the post-9/11 veteran population provide insights 
into student veterans and service members (Ramchand et al., 2010). Using meta-analysis strategies, 
Ramchand et al. (2010) found the PTS prevalence rate among the general population of Operation Iraqi 
Freedom (OIF) and Operation Enduring Freedom (OEF) veterans to be within the range of 5% to 20%, 
depending on the type of screening instrument used, the sampling procedures, and the definition of PTS 
used by the investigators. Milliken et al. (2007) found that 10.3% of OIF/OEF active-duty veterans and 
13.0% of National Guard and Reservists were at risk of depression six months after returning from 
deployment. They also reported that 16.7% of OIF/OEF active-duty veterans and 24.5% of National 
Guard and Reservist veterans were at risk for PTS six months after returning from deployment. Over one 
fourth (27.1%) of OIF/OEF active-duty veterans and over one third (35.5%) of National Guard and 
Reservist veterans were found to be at mental health risk, defined by the authors as being at risk for at 
least one of the following: depression, PTS, suicidal ideation, interpersonal conflict, or aggressive 
ideation. These rates of PTS among the post-9/11 veteran population are higher than the rates found in 
college students generally (9.0%) by Read, Ouimette, White, Colder, and Farrow (2011), but they are 
nearly even with the reported college-student prevalence of depression of 12.1% (American College 
Health Association, 2014). 

A lack of adequate research on psychological distress among the student veteran population 
makes it difficult to report how PTS and depression affect student veterans’ persistence rates. Ackerman 
et al. (2009) found that student veterans with PTS reported more difficulty with concentration and focus, 
thus affecting their academic performance, which may affect their persistence. Yet, it is logical to 
presume that the effects psychological distress has on the nonveteran student population will also be 
present among student veterans. Psychological stressors among nonveteran students can have a negative 
effect on their college persistence, as evidenced by recent studies that have explored the effects of 
psychological stressors on postsecondary retention, which can provide further insight on the effects of 
psychological distress on student veterans’ retention (Duncan, 2000; Kiuhara & Huefner, 2008; 
Megivem, Pellerito, & Mowbray, 2003; Smyth, Hockemeyer, Heron, Wonderlich, & Pennebaker, 2008; 
Thompson, 2007). Megivem et al. (2003) found a link between academic performance and psychological 
symptoms, which also subsequently led to some students withdrawing from postsecondary institutions. 

Duncan (2000) found an association between college students who report post-traumatic 
symptoms and their withdrawal from college. The study found a significant interaction (p < .01) between 
traumatic stress and PTS as related to remaining in or withdrawing from college. PTS symptoms present 
at the second week of a student’s freshman year were significantly related (p < .05) to that student’s 
college attendance in his or her fourth year, and students who experienced either more or stronger 
symptoms were less likely to be enrolled by their fourth year. Also, the number of students enrolled who 
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reported PTS symptoms present at the second week consistently decreased with each successive semester; 
the greatest decrease occurred at the end of the first year of school. Therefore, student veterans with 
psychological distress may have a lower probability of persisting and completing their degrees. 

Students with psychological distress are not new to postsecondary institutions. Postsecondary 
institutions have developed on-campus services and programs designed to support members of their 
student bodies who are experiencing psychological distress and research provided evidence that these 
campus program and services help mitigate some of the negative effects of psychological distress 
(DeStefano, Mellott, & Petersen, 2001; Turner & Berry, 2000; Wilson, Mason, & Ewing, 1997). 
DeStefano et al. (2001) found that on-campus counseling services increased the academic, social, and 
emotional adjustments of at-risk students to the point where there were no significant differences (p > .05) 
between the at-risk group and the control group. Wilson et al. (1997) reported that students who received 
counseling had 14% higher retention rates than their uncounseled peers. Turner and Berry (2000) also 
found that the postsecondary completion rates between students who received counseling and the general 
student population were equal. Considering that students in need of counseling are at risk of withdrawing 
from college, this finding illustrates the potential benefits of mental health counseling to enable students 
to adjust positively to college as well as persist in their pursuit of a college degree. 

Students will only receive the benefits of these programs and services if they use them. In some 
cases, the school relies on a central point of contact trained for and dedicated to counseling veterans 
(Cook & Kim, 2009); however, this method is inefficient and relies on the presumption that the student 
veteran will seek out the point of contact for help. Alternatively, faculty and campus-wide staff have more 
contact with student veterans and can provide referrals to on-campus support services for them, yet many 
colleges and universities do not provide training on veteran or military culture for their faculty or staff 
(Ryan, Carlstrom, Hughey, & Harris, 2011), thereby limiting the ability of faculty and staff to provide 
help and referrals to student veterans and hindering student veterans’ use of potentially beneficial campus 
support services. 

Literature Review 


Gatekeeper Training 

Gatekeeper training involves teaching individuals to recognize the signs of psychological distress 
in others and to be able to approach and talk with those they are concerned about for the purpose of 
making a referral to mental health support services. Internationally, the United Nations has recommended 
that gatekeeper training be considered an important part in implementing an effective strategy to prevent 
suicide (United Nations Department for Policy Coordination and Sustainable Development, 1996). In this 
study, college and university faculty, staff, and administrators completed an online gatekeeper training 
simulation where they were taught to become aware of the signs of psychological distress in student 
veterans, such as those seen in PTS, substance abuse, anxiety disorders, depression, and thoughts of 
suicide. Additionally, the training taught them how to engage a student veteran they are concerned about 
in a conversation, and if necessary, motivate him or her to seek mental health support services. 

Many gatekeeper training programs have demonstrated increases in knowledge and self- efficacy, 
but relatively few studies have examined changes in gatekeeper behaviors (Aseltine & DeMartino, 2004; 
Cross, Matthieu, Lezine, & Knox, 2010; Isaac et al., 2009; Kalafat & Gagliano, 1996; Mann et al., 2005; 
Schurtz, Cerel, & Rodgers, 2010; Wyman et al., 2008). Some studies examined the impact of gatekeeper 
training on knowledge, attitudes, and gatekeeper behaviors (Albright, 2013; Albright et al., 2012; Cross et 
al., 2011) as well as self-disclosed suicide attempts (Aseltine & DeMartino, 2004; Knox, Litts, Talcott, 
Feig, & Caine, 2003; Eggert, Thompson, Randell, & Pike, 2002). 

The majority of gatekeeper training programs do not employ active-learning strategies such as 
role rehearsal, in which learners practice gatekeeper skills by role-playing with an individual who is 
portraying himself or herself as being a distressed or suicidal person, to aid skill development in the 
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learner (Cross et al., 2011). Active learning involves learning by doing as opposed to solely listening. 
This process can involve reading, discussing, problem solving, role-playing, and participating in learning 
groups, games, simulations, and other activities that impact learner knowledge, skills, and attitudes 
(Bonwell & Eison, 1991). There are hundreds of journal articles including meta-analytic studies that 
examine the impact of active-learning strategies as applied to multiple disciplines; thus, to stay within the 
framework of this study, we are focusing on role-play as the main active-learning strategy. 

Researchers are exploring the implications of using role-plays during gatekeeper trainings to 
allow people the opportunity to practice the new skills. Cross et al. (2011) found that the use of role-plays 
or behavioral rehearsal resulted in higher overall gatekeeper skills immediately following training and at a 
3-month follow-up compared to those that did not participate in a role-play; however there was a 
significant reduction in both groups’ scores at the 3-month follow-up. 

Effective role-plays must have the proper context and simulate narratives that represent real- 
world scenarios. Otherwise the learning process is limited, and there may be a decreased likelihood to 
return for further training. To approximate a real-world experience, Cross et al. (2011) use trained 
experimenters in conducting face-to-face role-plays to train gatekeepers. 

Online role-play training simulation models were found to be an effective tool to engage military 
families in taking an active role in motivating veterans to seek help at a Veterans Administration (VA) 
hospital (Albright et al., 2011). Further support of the impact of this type of active learning used to train 
gatekeepers is seen in a meta-analytic study in which a total of 12,410 middle- and high-school educators, 
college faculty, staff, and students completed one of five online role-play training simulations (Albright et 
al., 2014b). Baseline, post-training, and 3-month follow-up data was collected via the Gatekeeper 
Behavior Scale (GBS), which measured gatekeeper attitudes of preparedness, likelihood or behavioral 
intent, and self-efficacy to engage in gatekeeper behaviors. Baseline and follow-up gatekeeper behaviors 
were also collected and included the self-reported number of students identified, approached, and referred 
to mental health support services. Composite effect size for pre- and posttraining measures of 
preparedness, likelihood, and self-efficacy to engage in gatekeeper behaviors was large at 0.72. Effect 
sizes comparing pretraining to posttraining preparedness was 0.70, likelihood was 0.35, and self-efficacy 
was 0.42. Changes in gatekeeper behaviors, including increases in the number of distressed students 
identified, approached, and referred to mental health support services, was 0.21. This data supports the 
use of the game-based role-play simulations using virtual humans to train gatekeepers. 

The gatekeeper training simulation used in this study is called Veterans on Campus (VOC) and 
was developed by a vendor, Kognito, with input from veteran services organizations, nationally 
recognized mental health experts, and over 75 faculty and student veterans in the United States. 
Introduced in 2012, VOC is listed in Section 111 of the Suicide Prevention Resource Center of the 
American Foundation for Suicide Prevention Best Practices Registry for Suicide Prevention (Suicide 
Prevention Resource Center, 2014). Section 111: Adherence to Standards lists suicide prevention programs 
and practices whose content has been reviewed for accuracy, likelihood of meeting objectives, and 
adherence to program design standards. 

Training Simulations 

VOC is a 30-minute online training simulation designed to educate faculty and staff about (1) the 
unique value student veterans bring to campus, (2) obstacles veterans may face in their pursuit of a 
college degree, (3) effective techniques for managing classroom discussions around topics that may be 
sensitive to veterans, and (4) best practices for connecting student veterans exhibiting signs of 
psychological distress with appropriate support services. VOC is built around a series of mini¬ 
conversation games where users interact with intelligent, fully animated, and emotionally responsive 
virtual student veterans experiencing a challenging situation or psychological distress. These virtual 
humans possess their own personalities and memories and adapt their behaviors to the learners’ 
conversation tactics throughout the role-play to provide the player with a highly realistic experience, as if 
they were interacting with real student veterans. Within these conversations, learners practiced using 


Online Learning - Vol. 19 Issue No. 1 (2015) 


51 


motivational interviewing (Ml) skills in role-plays. A virtual coach provides ongoing positive feedback 
when the learner employs effective Ml conversation strategies and corrective feedback in response to 
non-Ml or poor conversation choices, such as being judgmental or critical, or attempting to diagnose a 
problem. Additionally, the virtual veterans that users role-play with react verbally and nonverbally to the 
conversation tactics that learner selects, thus providing immediate feedback to the learner. The participant 
completes the training by successfully helping each veteran resolve a challenge he or she is facing due to 
individual transitions that result in an effective referral to support services. 

Motivational Interviewing 

In this study participants learn how to employ a set of MI techniques originally designed by 
clinical psychologists for use in counseling sessions with problem drinkers (Miller, 1983). Ml is a goal- 
oriented, client-centered counseling approach designed to actively engage clients in identifying their 
problems and to increase their intrinsic motivation to change their behavior (Miller & Rollnick, 2012). 
Numerous meta-analyses have demonstrated the efficacy of MI in a wide variety of clinical contexts, 
including weight loss, smoking cessation, adolescent substance abuse, problem drinking, and primary care 
(Armstrong et al., 2011; Burke, Arkowitz, & Menchola, 2003; Heckman, Egleston, & Hofmann 2010; 
Jensen et al., 2011; Lai, Cahill, Qin, & Tang, 2010; Lundahl & Burke, 2009; Rubak, Sandbank, Lauritzen, 
& Christensen, 2005; VanBuskirk & Wetherell, 2013; Vasilaki, Hosier, & Cox, 2006). The VOC training 
integrated the four core Ml skills, which include (1) asking open-ended questions, (2) providing 
affirmation, (3) reflective listening (listening closely and periodically confirming comprehension), and (4) 
summarizing client self-assessments. These techniques are associated with effective therapy and are 
thought to encourage strong rapport and to support behavioral change (Miller & Rollnick, 2012). 

Methods 

This quasi-experimental post-hoc study examines the impact of the VOC training simulation on 
758 faculty, staff, and administrators from 20 four-year colleges or universities and four community 
colleges. 

The Training 

To complete the training, users enter an online environment and engage in simulated role-play 
conversations with three emotionally responsive virtual student veterans. Participants practice role-plays 
with each simulated veteran and receive ongoing corrective or reinforcing feedback from a virtual coach. 
The participant completes the training by helping each veteran successfully resolve a challenge he or she 
is facing due to his or her transition. In this process, students learn to use Ml tactics such as asking open- 
ended questions and reflective listening to build trust and motivate the virtual veteran to seek help. 

Procedure and Measurers 

All participants were recruited by e-mail, mailbox flyer, poster, or a colleague. Campus-wide e- 
mails were from the chancellor, provost, vice president of student services, counseling center director, and 
campus-wide stakeholders, such as the veteran’s resource center or student veteran’s organizations, such 
as chapters of the Students Veterans of America. Before beginning the study, participants were informed 
that their participation was voluntary and that there would be no negative consequences if they decided 
not to participate. They were also informed that all identifying information and answers would be 
confidential and that their results would be combined with responses from other survey participants and 
might be presented at scientific or medical meetings or published in scientific journals. Lastly, they were 
informed that by submitting their answers to the survey, they were giving their consent and agreeing that 
they had read and understood the nature of their participation. 
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Gatekeeper Behaviors 
Table 1 Gatekeeper Behavior Scale 


Subscale 

Number 

Item 

Response scale 

Preparedness 

How would you rate your preparedness to: 



1 

Recognize when a student veteran’s behavior is a 
sign of psychological distress 



2 

Recognize when a student veteran’s physical 
appearance is a sign of psychological distress 

1- Very low 


3 

4 

Discuss with a student veteran your concern 
about the signs of psychological distress they are 
exhibiting 

Motivate student veterans exhibiting signs of 
psychological stress to seek help 

2- Low 

3- Medium 

4- High 

5- Very high 


5 

Recommend mental health support services (such 
as the counseling center) to a student veteran 
exhibiting signs of psychological distress 


Likelihood 

6 

7 

How likely are you to discuss your concerns with 
a student veteran exhibiting signs of 
psychological distress? 

How likely are you to recommend mental 
health/support services (such as the counseling 
center) to a student veteran exhibiting signs of 
psychological distress? 

1- Very unlikely 

2- Unlikely 

3- Likely 

4- Very likely 

Self-Efficacy 

Please rate how much you agree/disagree with the 
following statements: 



8 

9 

1 feel confident in my ability to discuss my 
concern with a student veteran exhibiting signs of 
psychological distress. 

1 feel confident in my ability to recommend 
mental health support services to a student 
veteran exhibiting signs of psychological distress. 

1 - Strongly disagree 

2- Disagree 

3- Agree 

4- Strongly agree 


10 

1 feel confident that 1 know where to refer a 
student veteran for mental health support. 



11 

1 feel confident in my ability to assist a student 
veteran seeking help. 



There were three surveys used in this study, which included a survey administered just before 
training to collect baseline measures, a survey administered immediately after the training, and a 3-month 
follow-up survey to assess durability of learning. All or some of the surveys included measures of (1) 
gatekeeper attitudes, (2) gatekeeper behaviors, (3) military cultural competency, (4) means efficacy, and 
(5) demographics. 
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Gatekeeper Attitudes 

All three surveys included the 11-item Gatekeeper Behavior Scale (GBS), a validated assessment 
tool used to determine the impact of online virtual-human gatekeeper training simulations (Albright et al., 
2014). The GBS’s items measure changes in attitudes and intentions by assessing three dimensions or 
subscales: learner preparedness and likelihood and self-efficacy to engage in gatekeeper behaviors. The 
preparedness composite comprises five items that indicate how prepared the learner was to (1) identify a 
student veteran in psychological distress as evidenced by his or her behavior, (2) identify a student 
veteran in psychological distress as evidenced by his or her appearance, (3) discuss his or her concern 
with the student veteran, (4) motivate a student veteran in psychological distress to seek help, and (5) 
recommend mental health support services. The second is the likelihood or behavioral intent that included 
two items on how likely the learner is to (1) discuss concerns and (2) recommend mental health services. 
The third construct is self-efficacy or confidence in one’s ability to (1) discuss one’s concern with a 
student veteran in psychological distress, (2) assist a student veteran seeking help, (3) recommend mental 
health support services, and (4) know where to refer student veterans for support services. The GBS’s 
construct and content validity were assessed via confirmatory factor analysis and had internal reliabilities 
of .95 for preparedness, .85 for likelihood, and .94 for self-efficacy. In this current study, all GBS items 
that contained the word “student” had veteran added so that they read “student veteran,” and there was no 
data examining the impact on validity because of this change (see Table 1 for the GBS). 

The pretraining and 3-month follow-up surveys contained four items that assessed gatekeeper 
behaviors. The phrase “Over the past two academic months how many student veterans have you” was 
followed by the four items: (1) spoken with about their military service, (2) been concerned about due to 
psychological distress, (3) been approached by to talk to about their concern, and (4) referred to support 
services. Gatekeeper behaviors were also measured in the 3-month follow-up survey, in which users were 
asked to respond on a 4-point Likert scale from “strongly agree” to “strongly disagree” with the statement 
“As a result of taking the VOC course there has been an increase in the number of student veterans that 1 
have” (1) recognized as exhibiting signs of psychological distress, (2) approached to discuss my concern 
about their psychological distress, (3) referred for mental health support, and (4) referred to non-mental- 
health campus offices. 

Military Cultural Competency 

The pretraining, posttraining, and follow-up surveys contained five items that measured military 
cultural competency (MCC). MCC is comprised of the knowledge, skills, and attitudes necessary to 
understand the veteran population and their families (Gleeson & Hemmer, 2014). The authors point out 
the importance of healthcare providers developing MCC to be more effective in helping veteran 
populations and their families. There are no validated measures of MCC in higher educational 
institutions; thus, we developed five items where participants were asked about (1) how prepared they 
were to talk with a student veteran about his or her military service, (2) knowledge about the common 
challenges facing the student veteran population, (3) preparedness to manage classroom discussions 
around veteran-sensitive issues, (4) comfort in talking to a student veteran about his or her military 
service, and (5) preparedness to refer student veterans to appropriate campus offices. This last item was 
included because there are many resources within higher education that a veteran can be referred to 
beyond a mental health counseling center, such as career services, a disabilities center, a student veteran 
association, and so on. Knowing the potential needs of student veterans and understanding their life 
circumstances is part of MCC. Response options were set on a 4-point Likert scale that ranged from 
“strongly disagree” to “strongly agree,” except for the preparedness to manage classroom discussions 
around veteran sensitive issues, which was on a 5-point scale ranging from “very low” to “very high.” 

Means Efficacy 

Means efficacy is defined by Eden, Ganzach, Flumin-Granat, and Zigman (2010) as an 
individual’s belief in the utility of the tools available for performing a job and is correlated with changes 
in behavior. Means efficacy items were only in the posttraining survey and were based on a 5-point Likert 
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scale that ranged from “not at all or to a very little extent” to “to a very great extent” and included 
measures of the training’s (1) usefulness, (2) ease of use, (3) well constructedness (“likely to help 
you with student veterans in psychological distress”), (4) “being based on scenarios that are 
relevant to you and student veterans,” and (5) “aid you in getting timely help to your veteran 
student.” 


Results 


Demographics 

Participants who responded to the demographic questions showed that the average age was 
43.8 years (SD = 13.4) with 67% female. In terms of race, 69% responded as white/Caucasian, 18% 
Hispanic or Latino, 8% black or African American, 4% Asian, and 1% American Indian/Alaskan 
Native. The average number of years worked at a college/university was 10.2 (SD = 8.6), and roles 
included full-time faculty (14%), part-time faculty (19%), staff member (65%), or resident 
advisor/student leader (2%). A total of 22.8% of participants stated they had received prior 
gatekeeper training, with 5.4% stating that they had been trained as a mental health practitioner. 

GBS Results 


Preparedness 

A repeated-measures analysis of variance (ANOVA) revealed statistically significant 
increases in participants’ GBS preparedness composite score to recognize, discuss, motivate, and 
help student veterans in psychological distress between pretraining, posttraining, and follow-up 
scores F( 2,758) = 921A, p < .01. Bonferroni’s correction indicated that the posttraining and 3- 
month follow-up scores were both significantly higher when compared to pretraining baseline 
scores. Follow-up scores were statistically significantly lower than posttraining scores (but still 
statistically significantly higher compared to baseline), indicating some reduction in preparedness 
over time (see Table 2). 

Likelihood 

A repeated-measures ANOVA showed statistically significant increases in participants’ 
GBS composite scores of likelihood to help student veterans in psychological distress between 
pretraining, posttraining, and follow-up scores F( 2, 742) = 228.0, p < .01. 

Bonferroni’s correction indicated that the posttraining and 3-month follow-up scores were 
both statistically significantly higher than pretraining baseline scores. Follow-up scores were 
statistically significantly lower than posttraining scores (though still significantly higher than 
baseline), indicating some reduction in preparedness over time (see Table 2). 


Table 2 Comparison of GBS, Military Cultural Competency and Gatekeeper Behavior Measures Over Time 


Composite measure 

N 

Pre 

M (SD) 

Post 

M(SD) 

Follow-up 

M (SD) 

F t 

Preparedness 

758 

2.94 (.85) 

3.87 (.67) 

3.78 (.66) 

927.35* - 

Likelihood 

743 

2.94 (.71) 

3.40 (.56) 

3.28 (.59) 

287.0* - 

Self-efficacy 

726 

2.70 (.65) 

3.30 (.46) 

3.17 (.50) 

520.7* - 

Military cul. comp. 

736 

2.92 (.62) 

3.33 (.48) 

3.27 (.50) 

274.0* - 

Behaviors 1 

258 

1.93 (.54) 

- 

2.75 (.11) 

1.41 


Note. Only those who indicated having veterans in their classrooms were included.*/? < .01 
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Self-Efficacy 

The third repeated-measures ANOVA showed statistically significant increases in participants’ 
GBS composite scores of self-efficacy with regard to helping student veterans in psychological distress 
between pretraining, posttraining, and follow-up scores F( 2, 725) = 520.37, p < .01. Follow-up scores 
were statistically significantly lower than posttraining scores (but still statistically significantly higher 
than baseline), indicating some reduction in self-efficacy over time (see Table 2). 

Gatekeeper Behaviors 

To investigate changes in gatekeeper behaviors from baseline to 3-month follow-up, we 
combined into a composite variable the four behavior items that included the self-reported number of 
student veterans that participants (1) had spoken with about their military service, (2) were concerned 
about due to psychological distress, (3) had approached to talk to about their concern, and (4) had referred 
to support services. The pretraining and follow-up composite scores for those participants that reported 
that they had a student veteran in their classroom (N = 200) were compared using a paired samples t-test. 
The result indicated the composite behavior score at 3-month follow-up compared to before VOC training 
was trending toward a statistically significant increase, 1(257) = 1.41 ,p = .160 (see Table 2). 

Ultimately, the impact of training comes down to the answer to the question, does the training 
increase gatekeeper behaviors? Thus, to further examine behavior change, we included Likert scale items 
in the 3-month follow-up survey in which participants were asked to respond to the statement “As a result 
of taking the VOC course there has been an increase in the number of student veterans that 1 have.” 
Results showed that (1) 52.5% agreed or strongly agreed with the statement that VOC increased 
recognition of those student veterans exhibiting signs of psychological distress, (2) 48% agreed or 
strongly agreed that the training increased the number of student veterans approached to discuss their 
concern about their psychological distress, (3) 45.5% agreed or strongly agreed that there were increases 
in the number of student veterans referred for mental health support services, and (4) 51% agreed or 
strongly agreed with the statement that VOC increased referrals. Responses to these items showed an 
increase in gatekeeper behaviors; however, the measures are self-reported, and validity and reliability 
have not been established. 

Military Cultural Competency 

MCC was administered in the pretraining, posttraining, and follow-up surveys, and was 
comprised of five Likert scale items in which participants reported on (1) how prepared they were to talk 
with a student veteran about his or her military service, (2) knowledge about the common challenges 
facing the student veteran population, (3) preparedness to manage classroom discussions around veteran- 
sensitive issues, (4) comfort in talking to a student veteran about his or her military service, and (5) 
preparedness to refer student veterans to appropriate campus offices. Reponses to these items were used 
to form a composite score and a repeated-measures ANOVA showed that there was statistically 
significant increases in the composite MCC score. Pre, post, and follow-up scores F (2, 735) = 274.0, p < 
.01. Bonferroni’s correction indicated that the posttraining and 3-month follow-up scores were 
statistically significantly higher than pretraining scores. Follow-up scores were statistically significantly 
lower than posttraining scores yet remained statistically significantly higher than baseline, indicating 
some reduction in MCC over time (see Table 2). 

The pretraining, posttraining, and follow-up measures from the composite scores drawn from the 
GBS, military cultural competency, and gatekeeper behavior can be seen in Figure 1. 

Means Efficacy 

All measures of means efficacy were high for the trainings and rated it as being a useful tool, well 
constructed, easy to use, “likely to help the learner with a student in psychological distress,” based on 
scenarios that are relevant and aiding the learner in getting timely help for the student (see Table 3). 
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Reaction data also showed that 99% of participants rated the course “good” to “excellent,” 98% 
would recommend VOC to other faculty and staff, and 96% stated that all faculty, staff, and 
administrators in their academic institution should take this course. Lastly, 38% of participants either 
agreed or strongly agreed that as a result of the training there has been an increase in the number of 
conversations they have had with other faculty, staff, and/or administrators regarding student veterans 
they are concerned about. 


Discussion 

This study provides support for using the online VOC role-play training simulation to increase 
gatekeeper skills and MCC in university and college faculty, staff, and administrators to support student 
veterans. To investigate whether skills were retained over time, we compared outcome measures collected 
at baseline and 3-month follow-up for the gatekeeper composite variables of preparedness, likelihood or 
behavioral intent, and self-efficacy to help student veterans, as well as MCC. All three gatekeeper 
composite variables and MCC remained statistically significantly higher when comparing baseline to 3- 
month follow-up, even though there was an expected decline in these scores from posttraining to follow¬ 
up. Thus, the analyses imply that the training has a lasting effect on gatekeeper skills and MCC, yet the 
decline in training effect also implies that it would be useful to have refresher training at periodic 
intervals, such as by including this training with other annual training for faculty, staff, and 
administrators. The composite gatekeeper behavior score increased and was trending toward significant 
levels from baseline to follow-up. Further support for increases in gatekeeper behavior were seen when 
45% of participants who had at least one veteran in their class stated they made a referral of a student 
veteran to mental health services and 51% to non-mental-health campus services. Lastly, means efficacy 
data was very high, indicating that users believed in the effectiveness of the tool, which has been shown 
to impact behavior. 
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Table 3 Means Efficacy Course Ratings 



To a very 
great extent 

To a great 
extent 

To some 
extent 

To a little 
extent 

Not at all or to a 
very little extent 

A useful tool 

33.5% 

45.1% 

19.4% 

1.7% 

0.0% 

Well constructed 

41.0% 

47.2% 

10.1% 

1.4% 

0.0% 

Easy to use 

55.1% 

37.6% 

5.1% 

1.0% 

0.0% 

Likely to help you with student 

veterans in psychological 
distress 

33.5% 

42.0% 

19.7% 

3.8% 

0.3% 

Based on scenarios that are 

relevant to you and student 
veterans 

35.6% 

42.8% 

16.8% 

3.4% 

0.7% 

Aid you in getting timely help 
to your veteran student 

33.5% 

43.9% 

17.7% 

3.2% 

0.4% 


Comments and Limitations 


Comments 

Data showing statistically significant and sustained increases in the GBS and MCC, coupled with 
self-reported increases in gatekeeper behaviors and high means efficacy, strongly suggest that VOC is an 
effective tool to train faculty and staff to better support student veterans. This includes understanding the 
common challenges student veterans face, talking to them about their military service, and managing 
classroom discussions around veteran-sensitive issues. The training is also effective in developing 
learners’ gatekeeper skills, including recognizing when a student veteran’s appearance or behavior is of 
concern, such as in depression, in which the veteran may look disheveled, stop attending class, or be 
chronically late, disengaged, or exhibit poor academic performance. These indicators can overlap into 
PTS, in which a student veteran might find it difficult to concentrate due to being easily distracted, easy to 
anger, or seeming anxious and/or withdrawn. These and other indicators show that the student veteran 
may be in psychological distress, and knowing how to identify, approach, talk to, and if necessary 
motivate him or her to seek appropriate help and support is paramount. 

Online virtual environments also offer the ability to explore situations that would be stressful in 
real life (Mantovani, Castelnuovo, Gaggioli, & Riva, 2003). For example, one study reported that medical 
students found face-to-face role-play embarrassing and stressful compared to role-play in Second Life 
(Hamilton et al., 2014). Similarly, although socially anxious persons disclose less personal information in 
normal face-to-face interactions than do less anxious persons (Reno & Kenny, 1992), several studies 
reported that socially anxious participants disclosed significantly more infonnation when communicating 
with virtual humans (Kang & Gratch, 2010; Lucas, Gratch, King, & Morency, 2014). 

Beyond the outcome data from this study, the authors feel that there are several important 
advantages that support utilizing the online role-play gatekeeper simulation model as opposed to face-to- 
face training. The first is that role-playing with virtual humans may reduce situational factors that 
compromise the effectiveness of face-to-face role-plays, such as perfonning in front of peers, instructors, 
and other role-players, which can cause embarrassment or social evaluative threat (Nestel & Tierney, 
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2007). Both negative emotions in general, and social evaluative threat in particular, are known to impede 
cognitive performance (Baumeister, Twenge, & Nuss, 2002; Payne et al., 2006). Another advantage is 
that once a virtual gatekeeper simulation is developed, it is sustainable. There are no costs for training, 
hiring, and maintaining professional instructors, which is especially onerous if one is scaling up to 
implement training on a state or national level. Additionally, this online training eliminates the need for 
travel expenses and the cost of participants being pulled away from their work, especially in rural areas. 

Another advantage is being able to provide realistic online role-plays to support Ml skills 
building that integrate continual feedback in a safe and confidential learner environment, such as in the 
privacy of one’s office or home, with high fidelity. Training fidelity refers to the quality of delivery 
across all trainers. To ensure high fidelity, the learning experience must be standardized to each 
participant every time they enter a role-play, which means consistently delivering accurate knowledge, 
realistic and engaging role-plays, and appropriate feedback. VOC is game-based, so you are not 
depending on the skill and experience of the trainers and their knowledge of the population they are 
training. In the game-play, each user can have an entirely different experience depending on his or her 
choice of conversation tactic as he or she navigates through the role-plays, thus ensuring high fidelity. 
The high fidelity of VOC is in part due to a comprehensive and iterative development process used in 
constructing the simulation, which involved integrating the knowledge and experience of nationally 
recognized subject matter experts, and beta testing by many end-users and students. 

It is also important to note that once participants are trained, they must have easy access to 
information about where to refer the individual they are concerned about, or they may not be as likely to 
engage in gatekeeper behaviors. To address this concern, online training simulations such as VOC can 
easily direct participants with the location of campus, institutional, or local resources to link student 
veterans to needed support services. 

Finally, in the 3-month follow-up survey, we asked participants if there had been an increase in 
the number of conversations they had with other faculty, staff, or administrators regarding student 
veterans they were concerned about. Over one third reported increases in the number of conversations. 
This result may imply that as the number of people trained increases, so will conversations within an 
institution regarding understanding and supporting student-veteran mental health initiatives. This may 
have potentially profound implications because there might exist opportunities to impact overall 
institutional awareness, which will hopefully provide student veterans with the recognition and support 
they so richly earned and deserve. 

Limitations 

One limitation is that this field study, as do many others, utilized a quasi-experimental design that 
applied a post-hoc analysis. Future studies should employ a randomized control group design to examine 
the impact on outcome measures. Another limitation is that most participants self-selected into the study, 
meaning that individuals who elected to take the training may be disproportionately predisposed to 
assuming a “helper role.” This possible confound is found in most all gatekeeper training programs and is 
especially true for face-to-face programs where the learner must be at a physical location for a specific 
time/date. Further, changes in actual referrals to mental and non-mental-health support services were self- 
reported due to a number of concerns, including issues surrounding privacy of data. Future research 
should be aimed at documenting whether those student veterans who were referred actually followed 
through. Finally, because participants reported an increased number of conversations with their colleagues 
about student veterans and mental health, future studies should include a sociological approach to 
determine the impact of these types of gatekeeper training simulations on institutional culture. It is in the 
interest of national public health initiatives to better understand how technology can be leveraged to have 
an impact on early detection and referral to treatment of those who are in need of help. 
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